Tootfinder

This https://arxiv.org/abs/2302.05797 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Global Convergence Rate of Deep Equilibrium Models with General Activations
In a recent paper, Ling et al. investigated the over-parametrized Deep Equilibrium Model (DEQ) with ReLU activation. They proved that the gradient descent converges to a globally optimal solution for the quadratic loss function at a linear convergence rate. This paper shows that this fact still holds for DEQs with any generally bounded activation with bounded first and second derivatives. Since the new activation function is generally non-homogeneous, bounding the least eigenvalue of the Gram m…

The neural basis of a cognitive function that suppresses the generation of mental imagery: evidence from a functional magnetic resonance imaging study https://www.mdpi.com/2411-5150/8/2/18 "left posterior cingulate gyrus suppresses mental imagery generation"; implications for underst…

The Neural Basis of a Cognitive Function That Suppresses the Generation of Mental Imagery: Evidence from a Functional Magnetic Resonance Imaging Study
This study elucidated the brain regions associated with the perception-driven suppression of mental imagery generation by comparing brain activation in a picture observation condition with that in a positive imagery generation condition. The assumption was that mental imagery generation would be suppressed in the former condition but not in the latter. The results show significant activation of the left posterior cingulate gyrus (PCgG) in the former condition compared to in the latter condition…

Optimisation challenge for superconducting adiabatic neural network implementing XOR and OR boolean functions
D. S. Pashin, M. V. Bastrakova, D. A. Rybin, I. I. Soloviev, A. E. Schegolev, N. V. Klenov
https://arxiv.org/abs/2405.03521

Optimisation challenge for superconducting adiabatic neural network implementing XOR and OR boolean functions
In this article, we consider designs of simple analog artificial neural networks based on adiabatic Josephson cells with a sigmoid activation function. A new approach based on the gradient descent method is developed to adjust the circuit parameters, allowing efficient signal transmission between the network layers. The proposed solution is demonstrated on the example of the system implementing XOR and OR logical operations.

This https://arxiv.org/abs/2309.03966 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_qfi…

Fourier Neural Network Approximation of Transition Densities in Finance
This paper introduces FourNet, a novel single-layer feed-forward neural network (FFNN) method designed to approximate transition densities for which closed-form expressions of their Fourier transforms, i.e. characteristic functions, are available. A unique feature of FourNet lies in its use of a Gaussian activation function, enabling exact Fourier and inverse Fourier transformations and drawing analogies with the Gaussian mixture model. We mathematically establish FourNet's capacity to approxim…

This https://arxiv.org/abs/2310.09527 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

A discontinuous plane wave neural network method for Helmholtz equation and time-harmonic Maxwell's equations
In this paper we propose a {\it discontinuous} plane wave neural network (DPWNN) method with $hp-$refinement for approximately solving Helmholtz equation and time-harmonic Maxwell equations. In this method, we define a quadratic functional as in the plane wave least square (PWLS) method with $h-$refinement and introduce new discretization sets spanned by element-wise neural network functions with a single hidden layer, where the activation function on each element is chosen as a complex-valued …

Three Decades of Activations: A Comprehensive Survey of 400 Activation Functions for Neural Networks
Vladim\'ir Kunc, Ji\v{r}\'i Kl\'ema
https://arxiv.org/abs/2402.09092

Three Decades of Activations: A Comprehensive Survey of 400 Activation Functions for Neural Networks
Neural networks have proven to be a highly effective tool for solving complex problems in many areas of life. Recently, their importance and practical usability have further been reinforced with the advent of deep learning. One of the important conditions for the success of neural networks is the choice of an appropriate activation function introducing non-linearity into the model. Many types of these functions have been proposed in the literature in the past, but there is no single comprehensi…

Microscopic origin of twist-dependent electron transfer rate in bilayer graphene
Leonardo Coello Escalante, David T. Limmer
https://arxiv.org/abs/2405.00783

Microscopic origin of twist-dependent electron transfer rate in bilayer graphene
Using molecular simulation and continuum dielectric theory, we consider how electrochemical kinetics are modulated as a function of twist angle in bilayer graphene electrodes. By establishing an effective connection between twist angle and the screening length of charge carriers within the electrode, we investigate how tunable metallicity can result in modified statistics of the electron transfer energy gap. Constant potential molecular simulations show that the activation free energy for elect…

This https://arxiv.org/abs/2306.03303 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Global universal approximation of functional input maps on weighted spaces
We introduce so-called functional input neural networks defined on a possibly infinite dimensional weighted space with values also in a possibly infinite dimensional output space. To this end, we use an additive family to map the input weighted space to the hidden layer, on which a non-linear scalar activation function is applied to each neuron, and finally return the output via some linear readouts. Relying on Stone-Weierstrass theorems on weighted spaces, we can prove a global universal appro…

An Augmented Lagrangian Method for Training Recurrent Neural Networks
Yue Wang, Chao Zhang, Xiaojun Chen
https://arxiv.org/abs/2402.13687 https://

An Augmented Lagrangian Method for Training Recurrent Neural Networks
Recurrent Neural Networks (RNNs) are widely used to model sequential data in a wide range of areas, such as natural language processing, speech recognition, machine translation, and time series analysis. In this paper, we model the training process of RNNs with the ReLU activation function as a constrained optimization problem with a smooth nonconvex objective function and piecewise smooth nonconvex constraints. We prove that any feasible point of the optimization problem satisfies the no nonze…

Product of Tensors and Description of Networks
Luca Chiantini, Giuseppe Alessio D'Inverno, Sara Marziali
https://arxiv.org/abs/2402.06768 https://

Product of Tensors and Description of Networks
Any kind of network can be naturally represented by a Directed Acyclic Graph (DAG); additionally, activation functions can determine the reaction of each node of the network with respect to the signal(s) incoming. We study the characterization of the signal distribution in a network under the lens of tensor algebra. More specifically, we describe every activation function as tensor distributions with respect to the nodes, called \textit{activation tensors}. The distribution of the signal is enc…

This https://arxiv.org/abs/2402.10118 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csAR_…

Reusing Softmax Hardware Unit for GELU Computation in Transformers
Transformers have improved drastically the performance of natural language processing (NLP) and computer vision applications. The computation of transformers involves matrix multiplications and non-linear activation functions such as softmax and GELU (Gaussion Error Linear Unit) that are accelerated directly in hardware. Currently, function evaluation is done separately for each function and rarely allows for hardware reuse. To mitigate this problem, in this work, we map the computation of GELU…

Machine learning activation function parameterization of the occupation numbers for natural orbital functionals based on electron pairing approaches
L. Franco, I. A. Bonfil-Rivera, J. F. Huan Lew-Yee, M. Piris, J. M. del Campo, R. A. Vargas-Hern\'andez
https://arxiv.org/abs/2403.09463

Machine learning activation function parameterization of the occupation numbers for natural orbital functionals based on electron pairing approaches
Within the framework of natural orbital functional theory, having a convenient representation of the optimization function becomes critical for the computational performance of the calculation. Recognizing this, we propose an innovative parametrization of the occupation numbers that takes advantage of the electron-pairing approach used in Piris natural orbital functionals, through the adoption of the softmax function, a pivotal component in modern deep-learning models. Our approach not only ens…

This https://arxiv.org/abs/2310.12079 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Differential Equation Scaling Limits of Shaped and Unshaped Neural Networks
Recent analyses of neural networks with shaped activations (i.e. the activation function is scaled as the network size grows) have led to scaling limits described by differential equations. However, these results do not a priori tell us anything about "ordinary" unshaped networks, where the activation is unchanged as the network size grows. In this article, we find similar differential equation based asymptotic characterization for two types of unshaped networks. Firstly, we show that the fol…

Verifying message-passing neural networks via topology-based bounds tightening
Christopher Hojny, Shiqiang Zhang, Juan S. Campos, Ruth Misener
https://arxiv.org/abs/2402.13937

Verifying message-passing neural networks via topology-based bounds tightening
Since graph neural networks (GNNs) are often vulnerable to attack, we need to know when we can trust them. We develop a computationally effective approach towards providing robust certificates for message-passing neural networks (MPNNs) using a Rectified Linear Unit (ReLU) activation function. Because our work builds on mixed-integer optimization, it encodes a wide variety of subproblems, for example it admits (i) both adding and removing edges, (ii) both global and local budgets, and (iii) bot…

Reusing Softmax Hardware Unit for GELU Computation in Transformers
Christodoulos Peltekis, Kosmas Alexandridi, Giorgos Dimitrakopoulos
https://arxiv.org/abs/2402.10118

Reusing Softmax Hardware Unit for GELU Computation in Transformers
Transformers have improved drastically the performance of natural language processing (NLP) and computer vision applications. The computation of transformers involves matrix multiplications and non-linear activation functions such as softmax and GELU (Gaussion Error Linear Unit) that are accelerated directly in hardware. Currently, function evaluation is done separately for each function and rarely allows for hardware reuse. To mitigate this problem, in this work, we map the computation of GELU…

This https://arxiv.org/abs/2310.12079 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Differential Equation Scaling Limits of Shaped and Unshaped Neural Networks
Recent analyses of neural networks with shaped activations (i.e. the activation function is scaled as the network size grows) have led to scaling limits described by differential equations. However, these results do not a priori tell us anything about "ordinary" unshaped networks, where the activation is unchanged as the network size grows. In this article, we find similar differential equation based asymptotic characterization for two types of unshaped networks. Firstly, we show that the fol…

This https://arxiv.org/abs/2402.13092 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Contractivity of neural ODEs: an eigenvalue optimization problem
We propose a novel methodology to solve a key eigenvalue optimization problem which arises in the contractivity analysis of neural ODEs. When looking at contractivity properties of a one layer weight-tied neural ODE $\dot{u}(t)=σ(Au(t)+b)$ (with $u,b \in {\mathbb R}^n$, $A$ is a given $n \times n$ matrix, $σ: {\mathbb R} \to {\mathbb R}^+$ denotes an activation function and for a vector $z \in {\mathbb R}^n$, $σ(z) \in {\mathbb R}^n$ has to be interpreted entry-wise), we are led to study the…

This https://arxiv.org/abs/2305.00608 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Differentiable Neural Networks with RePU Activation: with Applications to Score Estimation and Isotonic Regression
We study the properties of differentiable neural networks activated by rectified power unit (RePU) functions. We show that the partial derivatives of RePU neural networks can be represented by RePUs mixed-activated networks and derive upper bounds for the complexity of the function class of derivatives of RePUs networks. We establish error bounds for simultaneously approximating $C^s$ smooth functions and their derivatives using RePU-activated deep neural networks. Furthermore, we derive improv…

Contractivity of neural ODEs: an eigenvalue optimization problem
Nicola Guglielmi, Arturo De Marinis, Anton Savastianov, Francesco Tudisco
https://arxiv.org/abs/2402.13092

Contractivity of neural ODEs: an eigenvalue optimization problem
We propose a novel methodology to solve a key eigenvalue optimization problem which arises in the contractivity analysis of neural ODEs. When looking at contractivity properties of a one layer weight-tied neural ODE $\dot{u}(t)=σ(Au(t)+b)$ (with $u,b \in {\mathbb R}^n$, $A$ is a given $n \times n$ matrix, $σ: {\mathbb R} \to {\mathbb R}^+$ denotes an activation function and for a vector $z \in {\mathbb R}^n$, $σ(z) \in {\mathbb R}^n$ has to be interpreted entry-wise), we are led to study the…

This https://arxiv.org/abs/2208.04284 has been replaced.
link: https://scholar.google.com/scholar?q=a

On Rademacher Complexity-based Generalization Bounds for Deep Learning
We show that the Rademacher complexity-based approach can generate non-vacuous generalisation bounds on Convolutional Neural Networks (CNNs) for classifying a small number of classes of images. The development of new Talagrand's contraction lemmas for high-dimensional mappings between function spaces and CNNs for general Lipschitz activation functions is a key technical contribution. Our results show that the Rademacher complexity does not depend on the network length for CNNs with some special…